Systems and methods for adaptive histopathology image unmixing

ABSTRACT

The present invention relates to systems and methods for adaptively optimizing broadband reference spectra for a multi-spectral image or adaptively optimizing reference colors for a bright-field image. The methods and systems of the present invention involve optimization techniques that are based on structures detected in an unmixed channel of the image, and involves detecting and segmenting structures from a channel, updating a reference matrix with signals estimated from the structures, subsequently unmixing the image using the updated reference matrix, and iteratively repeating the process until an optimized reference matrix is achieved.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. patent application Ser. No. 15/084,467filed Mar. 29, 2016, which is a continuation of International PatentApplication No. PCT/EP2014/070793 filed Sep. 29, 2014, which claimspriority to and the benefit of U.S. Provisional Patent Application No.61/884,974 filed Sep. 30, 2013, all of which prior applications areincorporated herein by reference as if set forth in their entirety.

FIELD OF INVENTION

The present invention relates to the unmixing of images, for example,histopathology images, in fluorescence microscopy or bright-fieldmicroscopy. More particularly, the present invention relates todynamically and adaptively refining reference spectra for amulti-spectral image or reference colors for the bright-field image.

BACKGROUND

In a multiplex fluorescent slide of a tissue specimen, different nucleiand tissue structures are simultaneously stained with specificfluorescent dyes, organic fluorescent dyes, or fluorescent counterstains, each of which fluoresces in a different spectral band, whilegenerally overlapping in the 400 nm-850 nm spectral range. Morerecently, quantum dots are widely used in immunofluorescence stainingfor the biomarkers of interest due to their intense and stablefluorescence. On a typical multiplex slide, a nuclei marker, forexample, a 4′,6-diamidino-2-phenylindole (DAPI) fluorescent stain (i.e.,a blue stain), is used along with the quantum dots. However, othernuclei counter stains may be used, such as, for example, otherfluorescent counter stains. The slide is then imaged using amulti-spectral imaging system (such as a fluorescent microscope systemthat is coupled to a camera or a scanner; or a whole slide scanner).Each channel of the imaging system corresponds to a spectral narrow-bandfilter. The multi-spectral image stack produced by the imaging system istherefore a mixture of the underlying biomarker expressions, which, insome instances, may be co-localized.

In brightfield unmixing, the nuclei and tissue structures are stainedwith hematoxylin and eosin (H&E) or IHC staining. The slide is thenscanned with the bright-field scanner equipped with CCD color camera andfinally the RGB image is acquired. Similarly to the multi-spectral imageanalysis, the reference colors of the RGB image are obtained from thescanned single stain bright-field images.

Taking the multi-spectral image unmixing as an example, to identify theindividual constituent fluorescent dyes for the biomarkers and theproportions they appear in the mixture, spectral unmixing is used todecompose each pixel of the multi-spectral image into a collection ofconstituent spectrum endmembers (or reference spectra) and the fractionsof their intensity contributions in the multi-spectral image from eachof them. The reference spectrum for a stain corresponds to the emissionspectral signature for the particular stain (e.g., fluorescent dye),when the stain is irradiated with spectra, for example, light of varyingexcitation wavelengths. The amount of endmember contribution is alsoreferred to as abundance, and corresponds to a pixel value in theunmixed image, for example the grayscale pixel values in the DAPI orquantum dot unmixed images.

Accurate spectral unmixing of fluorescent images is clinically importantbecause it is one of the key steps in multiplex histopathology imageanalysis. Several techniques have been proposed for spectral unmixing inthe field of remote sensing, for example. One popular approach issolving for the abundances given the reference spectra within thenon-negative least square (NNLS) framework. In this case, accurateestimation of the endmember contributions requires precise knowledgeabout the reference spectra. Unlike the applications in the domain ofremote sensing where different objects such as grass and rocks can beeasily identified from the scene, the biomarkers are often co-localizedin histopathology fluorescent images, therefore making it difficult toextract the endmember spectra from the image and solve for theabundances. While the narrow-band reference spectra for quantum dot ornanocrystal biomarkers can be precisely measured from single stainedcontrol slides, broadband signals such as DAPI and tissueauto-fluorescence (AF) are slide-specific and exhibit variation betweenimages and slide specimens. The broad-band spectra overlap with thenarrow-band spectra, making the accurate estimation of the quantum dotabundances (i.e., the unmixed images) even more difficult. In addition,part of the DAPI spectrum may be wrongly estimated as AF because thespectral signatures of DAPI and AF are similar to each other, which maylead to an erroneous estimation of the nuclear component.

To simultaneously estimate the reference spectra, as well as theendmember contributions, non-negative matrix factorization (NMF) is usedwidely for blind spectral unmixing. However, due to the non-linearity ofthe problem, this method is not guaranteed to converge to a physicallymeaningful and relevant solution. This is especially relevant for theDAPI estimation because it is possible for NMF to worsen with everyiteration for the same reason that the DAPI spectrum can be confusedwith other similar reference spectra, such as AF. NMF automaticallysolves for the reference spectral matrix, however the algorithm is notable to identify which reference spectrum corresponds to DAPI and whichone corresponds to AF. Additional frameworks proposed based on theorthogonality assumption (i.e., the assumption that the referencespectrum of DAPI is orthogonal to that of quantum dots) of the endmemberdo not yield meaningful results for real data. Accurate DAPI unmixing isof great clinical importance because it is the most common nuclearstain. Moreover, nuclei detection serves as a first step in digitalpathology image analysis, with further analysis tasks being based on thereliable identification of cell nuclei. Thus, there is a need forprecise unmixing results. In the case of bright-field images,hematoxylin plays an equivalently important role as DAPI inmulti-spectral images. Hence, the correct unmixing of hematoxylin isalso very important.

SUMMARY OF THE SUBJECT DISCLOSURE

The subject disclosure presents systems and methods for adaptivelyoptimizing the reference spectra for a multi-spectral image or referencecolors for bright-field image. Here, the updating of the broadbandspectra of the multi-spectral image is an example to explain theprocess, however, the same procedure can be applied to the updating ofthe reference colors of the bright-field images. This adaptiveoptimization is based on structures detected in an unmixed broadbandchannel of the image. Embodiments disclosed herein perform operationsincluding detecting and segmenting structures, such as nuclei from anunmixed image of broadband channel, tissue structures, and boundariesthereof, and updating a reference matrix with one or more broadbandsignals estimated from the identified structures and their surroundingregions. A confidence level for each structure may be determined basedon the structure's shape and grayscale intensity information andbiological criteria, with broadband signatures from only high-confidencestructures being used to update the reference spectra in the referencematrix. An updated reference spectra is estimated from thehigh-confidence structure regions by obtaining the pixel values from theoriginal multi-spectral image and using that to replace the existingreference spectra in the reference matrix. A subsequent unmixingoperation using the updated reference matrix yields improved results.The reference spectra updating and subsequent unmixing may beiteratively repeated until an optimized reference matrix is achieved.The optimization may be based on reaching a threshold number ofiterations, or convergence or stabilization of the reference matrix, orany combination thereof. The optimized reference matrix yields reliableunmixing of the hyper-spectral data that are superior when compared toexisting methods. The unmixing operations may be performed on the sameor different regions of the image, or the entire image repeatedly.Custom regions may be defined to enable analysis of images havingvarying broadband signatures. Separate operations may be executed inparallel on different regions, enabling efficient processing of largenumbers of multiplex fluorescent slides. In an exemplary embodiment ofthe present invention, a non-transitory computer-readable medium forstoring computer-executable instructions that are executed by aprocessor to perform operations is disclosed that involves utilizing areference matrix comprising an initial reference spectra or colors tounmix an image or regions of an image comprising a mixture of signals;estimating an updated reference spectra or colors for the image orregions of the image; and updating the reference matrix with the updatedreference spectra or reference colors; wherein the updated referencespectra or reference colors are used in a subsequent unmixing operation.In an exemplary embodiment of the present invention, the unmixedchannels may be one or more of a 4′,6-diamidino-2-phenylindole (DAPI)signal, or an autofluorescence signal, and hematoxylin (HTX) signals.

In one embodiment of the present invention, a system for image unmixinginvolves a processor; and a memory coupled to the processor, the memoryto store computer-executable instructions that, when executed by theprocessor, cause the processor to perform operations comprising:estimating a reference spectra for a stain from a highly ranked subsetof a plurality of structures detected from an unmixed signalcorresponding to that stain; and unmixing the image using the referencespectra; wherein the estimating and unmixing are repeated until anoptimal reference spectra is acquired.

In another embodiment of the present invention, a method is disclosedthat involves unmixing an image comprising a mixture of signals;

detecting a plurality of structures in the unmixed signal; ranking theplurality of structures in order of a confidence level; estimating areference spectra for a stain from a subset of structures among theplurality of structures that have a confidence level higher than athreshold confidence level; storing the reference spectra in a referencematrix associated with the image; and subsequently unmixing the imageusing the reference matrix.

In another embodiment of the present invention, a non-transitorycomputer-readable medium for storing computer-executable instructions isdisclosed that is executed by a processor to perform operations,including utilizing a reference matrix having an initial referencevector to unmix a first region of an image comprising a mixture ofsignals, and generate an unmixed first region of the image; estimatingupdated reference spectra for the unmixed first region and generating anupdated reference vector for the unmixed first region; and updating thereference matrix with the updated reference vector, wherein the updatedreference vector is used in a subsequent unmixing operation.

In yet another embodiment of the present invention, a system forunmixing an image is disclosed comprising a processor; and a memorycoupled to the processor, the memory stores computer-executableinstructions that, when executed by the processor, cause the processorto perform operations comprising: estimating reference vector for astain from a highly ranked subset of a plurality of structures detectedfrom an unmixed signal corresponding to the stain; and unmixing theimage using the reference spectra, wherein the estimating and unmixingare repeated until an optimal reference spectra is acquired.

In yet another embodiment of the present invention, a method forunmixing is disclosed, comprising: unmixing an image comprising amixture of signals and generating unmixed signals; detecting a pluralityof structures in the unmixed signals; ranking the plurality ofstructures in order of a confidence level; estimating a reference vectorfor a stain, shown in the image, from a subset of structures among theplurality of structures that have a confidence level higher than athreshold confidence level; storing the reference vector in a referencematrix associated with the image; and subsequently unmixing the imageusing the reference matrix.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a system for adaptive spectral unmixing, according to anexemplary embodiment of the present subject disclosure.

FIG. 2 shows a method for adaptive spectral unmixing, according to anexemplary embodiment of the present subject disclosure.

FIGS. 3A and 3B show an image divided into regions for adaptive spectralunmixing, according to an exemplary embodiment of the present subjectdisclosure.

FIG. 4 shows a method for detecting structures in a broadband signal ofan image, according to an exemplary embodiment of the present subjectdisclosure.

FIG. 5 shows a method for determining a confidence level of detectedstructures, according to an exemplary embodiment of the present subjectdisclosure.

FIG. 6 shows a nuclear ranking using ellipse fitting, according to anexemplary embodiment of the present subject disclosure.

FIG. 7 shows a median reference spectrum estimated from a plurality ofhighly-ranked structures, according to an exemplary embodiment of thesubject disclosure.

FIG. 8 shows a comparison of nuclear detection results using an initialreference spectra versus an updated reference spectra, according toexemplary embodiments of the subject disclosure.

DETAILED DESCRIPTION OF THE SUBJECT DISCLOSURE

The subject disclosure presents systems and methods for adaptivelyoptimizing the broadband reference spectra for an image comprising aplurality of fluorescent channels or updating the reference color for abright-field image. For each region of an image, we update the referencevector, i.e. a reference spectrum in the case of a fluorescence imageand a reference color in the case of a bright field image spectra andunmix the region till convergence. As an example application, thisdisclosure presents the details of the reference spectral refinementmethod for multi-spectral images, but the same technique can be appliedto the bright-field images.

Taking the fluorescent image as an example, this adaptive optimizationis based on structures detected in an unmixed broadband channel of theimage. For instance, a slide holding a sample material may be scannedusing a scanner coupled to a fluorescence or brightfield microscopesystem to generate a scanned image. The image is stored on acomputer-readable medium, and contains a mixture of several fluorescentor color channels, including one or more broadband channels, such as aDAPI channel. Exemplary embodiments disclosed herein detect nuclear andtissue regions from a broadband channel in an unmixed image and refinethe reference spectra with the broadband signatures of the biomarkerswithin these regions of the image. A linear spectral unmixing processsuch as a non-negative least-squares method shown herein may be utilizedto separate the component fluorescent channels. The image may comprise amixture of endmember spectra including DAPI and AF, in addition to oneor more quantum dots, as described above, that are initially unmixedusing an initial reference matrix. The initial reference spectra foreach marker may be retrieved from control data, a control image, or maybe estimated from the image under observation. For instance, the initialbroadband reference signal for the stain, or initial reference spectra,may be determined by observing a ubiquitous dispersion of a uniquesignature of the stain throughout the image or a region of the image,and identifying the signal based on a comparison of the stain'ssignature spectra emissions/output when illuminated with a knownsignature associated with the stain. The initial reference spectra for aparticular stain may be retrieved from a database containing one or morereference matrices. The initial reference spectra or reference colorsmay correspond to a reference matrix used in an unmixing operation.

Embodiments disclosed herein perform operations including observing theunmixed broadband signal, detecting and segmenting structures such asnuclei from the unmixed broadband signal channel, tissue structures, andboundaries thereof, and estimating updated reference spectra from thedetected structures. One or more broadband signals observed from thesestructures and their surrounding regions may be used as values in anupdated reference matrix that is used in a subsequent unmixingoperation. Moreover, shape and intensity information of the structuresin a region of the image, along with biological criteria, may be appliedin a determination of a confidence level for each structure. Thedetected structures may be scored to generate the confidence level, withreference spectra from only high-confidence structures being used in thesubsequent unmixing operation. Further embodiments disclosed hereinperform operations including ranking detected structures such as nucleibased on one or more criteria. For instance, a subset of the detectednuclei that are scored with a confidence level higher than a thresholdmay be accepted as true positives. These accepted structures areobserved to automatically estimate an updated reference spectra. Forinstance, the value of the updated reference spectrum from the selectednuclei may replace the initial DAPI reference spectrum in an updatedreference matrix. A subsequent unmixing operation using the updatedreference matrix therefore yields improved results.

The reference spectra estimation and subsequent unmixing may be repeatedand the reference spectra may be updated iteratively until an optimizedreference matrix is achieved. The subsequent unmixing may be performedon all the nuclei within the image or region thereof, or may beperformed on high-confidence nuclei, or separate subsets of nuclei, andany combination thereof. The determination of an optimized referencespectra may depend on a threshold number of iterations, or convergenceor stabilization of the reference matrix, or any combination thereof.The optimized reference matrix yields reliable unmixing of thehyper-spectral data that are superior when compared to existing methods.For instance, an unmixing operation using optimized DAPI referencespectra may generate sharper contours between nuclei and surroundingtissue.

Moreover, subsequent unmixing operations may be performed on the same ordifferent regions of the image, or the entire image repeatedly. Forinstance, a sparse grid (i.e., a grid having a large distance within anytwo vertices) may be imposed over the image, and structures within adistance of one or more grid vertices may be sampled for estimatedreference spectra. Custom regions may be defined based on structures orfeatures observed in the image, with separate optimized referencematrices per region enabling analysis of images having varying broadbandsignatures. Separate operations may be executed in parallel on differentregions, enabling efficient processing of large numbers of multiplexfluorescent slides. Subsequent unmixing operations may be performed onall structures within a region, or on separate subsets of structuresbased on a confidence level of the structures and surrounding regions,and other biological criteria.

FIG. 1 shows a system 100 for adaptive spectral unmixing, according toan exemplary embodiment of the present subject disclosure. System 100comprises a source 101 for generating an image, for example afluorescent image. Other types of images are also within the purview ofthe subject disclosure, such as an immunohistochemical (IHC)bright-field image with hematoxylin/eosin, and other stains. Forinstance, source 101 may be a fluorescence microscope generating afluorescent image, or a bright-field microscope generating an RGB image.Source 101 is in communication with a memory 103, which includes aplurality of processing modules or logical operations that are executedby processor 105 coupled to interface 107. For instance, a sample, suchas a biological specimen, may be mounted on a slide or other substrateor device for purposes of imaging by a microscope coupled to memory 103,with analysis of images of the specimen being performed by processor 105executing one or more of the plurality of modules stored on memory 103in accordance with the present disclosure. The analysis may be forpurposes of identification and study of the specimen. For instance, abiological or pathological system may study the specimen for biologicalinformation, such as the presence of proteins, protein fragments orother markers indicative of cancer or other disease, or for otherpurposes such as genomic DNA detection, messenger RNA detection, proteindetection, detection of viruses, detection of genes, or other.

The specimen may be stained by means of application of one or moredifferent stains that may contain one or more different quantum dots,fluorophore(s), or other stains. The number N of stains, for example,fluorophores that are applied to the specimen can vary. The fluorophoresmay comprise one or more nano-crystalline semiconductor fluorophores(i.e., quantum dots), each producing a peak luminescent response in adifferent range of wavelengths. Quantum dots are well known, and may becommercially available from Invitrogen Corp., Evident Technologies, andothers. For example, the specimen may be treated with several differentquantum dots, which respectively produce a peak luminescent response at565, 585, 605, and 655 nm. One or more of the fluorophores applied tothe specimen may be organic fluorophores 14 (e.g., DAPI, Texas Red),which are well known in the art, and are described in at leastcommonly-owned and assigned U.S. Pat. No. 8,290,236, the contents ofwhich are incorporated by reference herein in their entirety. Moreover,a typical specimen is processed in an automated staining/assay platformthat applies a stain containing quantum dots and/or organic fluorophoresto the specimen. There are a variety of commercial products on themarket suitable for use as the staining/assay platform, one examplebeing the Discovery™ automated slide staining platform of the assigneeVentana Medical Systems, Inc.

After preliminary tissue processing and staining, the specimen issupplied to an image acquisition module 111 to generate a digital imageof the specimen observed at source 101. Image acquisition module 111 maybe coupled to, for instance, a scanner or spectral camera that is usedfor imaging a slide containing a sample of a material stained with afluorescent stain and a light source for illuminating the specimen atwavelengths intended to produce a luminescent response from thefluorophores applied to the specimen. In the case of quantum dots, thelight source may be a broad spectrum light source. Alternatively, thelight source may comprise a narrow band light source such as a laser.The camera platform may also include a microscope having one or moreobjective lenses and a digital imager, as well as a set of spectralfilters. Other techniques for capturing images at different wavelengthsmay be used. Camera platforms suitable for imaging stained biologicalspecimens are known in the art and commercially available from companiessuch as Zeiss, Canon, Applied Spectral Imaging, and others, and suchplatforms are readily adaptable for use in the system, methods andapparatus of this subject disclosure. The image may be supplied tocomputer-readable medium 103, either via a cable connection between thesource 101 and computer 107, via a computer network, or using any othermedium that is commonly used to transfer digital information betweencomputers. The image may also be supplied over the network to a networkserver or database for storage and later retrieval by computer 107.Besides processor 105 and memory 103, computer 107 also includes userinput and output devices such as a keyboard, mouse, stylus, and adisplay/touchscreen. As will be explained in the following discussion,processor 105 executes modules stored on memory 103, performing analysisof the image, morphological processing of the image or image dataderived from such images, quantitative analysis, and display ofquantitative/graphical results to a user operating computer 107.

For instance, as described above, a slide, holding a specimen, isobserved at source 101, and a scanned image comprising a mixture ofseveral fluorescent channels including one or more broadband signals isgenerated at image acquisition module 111 as described above. Regionselection module 112 enables automated or manual delineation of theimage into one or more regions. This enables subsequent unmixingoperations to be performed on the same or different regions of theimage, enabling efficient processing of multiple regions of one image ormultiple images. For instance, a grid, such as a sparse grid may beimposed over an image, and structures within a distance of one or moregrid vertices may be sampled for estimated reference spectra, as furtherdescribed herein. The grid size may be optimized for speed or fordetailed processing, and may be defined by a user of the system. Customregions may be defined based on structures or features observed in theimage, with separate optimized references per region enabling analysisof images having varying broadband signatures. The custom regions may beautomatically determined based on image analysis, tissue heterogeneity,etc., or may be selected by the user. Separate operations may beexecuted in parallel on different regions, enabling efficient processingof large numbers of multiplex slides, for example, fluorescent slides. Aregion containing predominantly or purely broadband signals without anyquantum dot signals may be selected to determine an initial referencespectra or estimated reference spectra as described herein.

A spectral unmixing module 114 may be executed to unmix the image orselected regions of the image using a non-negative least-squares methodas shown herein for separating the component fluorescent channels. Theimage may comprise a mixture of endmember spectra including DAPI and AF,in addition to one or more quantum dots, as described above, that areinitially unmixed using initial reference spectra. The initial referencespectra may be retrieved from a control image, or may be estimated fromthe image under observation. The initial reference spectra may beretrieved from a database containing one or more reference matrices,such as reference spectra database 113. The initial reference spectramay be acquired from a reference matrix used in an unmixing operation.

The initial broadband reference signal, or initial reference signal, mayfurther be determined by observing a ubiquitous dispersion of a uniquesignature throughout the image or a region of the image. For instance, aubiquitously-dispersed broadband signal may be identified as a DAPIsignal based on a comparison of its signature with a known signaturefrom a known broadband signal (i.e. DAPI sampled from a control slide).As described above, one or more initial reference spectra may beacquired from a control image or determined from the image underobservation. The initial reference spectra may be estimated based upon acomparison of a measured broadband signature with known broadbandsignals stored in reference spectra database 113. One or more broadbandsignatures within the image may be recognized by its unique signatureand ubiquitous dispersion through the image. Certain regions of theimage may be determined to contain predominantly, or only, a broadbandsignal, such as autofluorescence, etc. The profile of these broadbandsignals along with their ubiquitous dispersion throughout the imagesupport an assumption that these recognized signals may be used asinitial reference spectra for unmixing. Moreover, upon determining acomponent signal having a broadband signature, the component signal maybe compared with known broadband signatures specific to, for examplesstains and/or the specimen being analyzed. For instance, a system foranatomical or clinical pathology may compare a scanned slide of a tissuespecimen with an image of a calibration slide containing similar tissuespecimens having known broadband signatures, to identify the broadbandsignals in the scanned image.

Spectral unmixing module 114 unmixes the component signals of the imageand/or regions thereof, wherein the weight of each stain is computed atevery image pixel, given the spectral signature at each pixel, usinginitial reference spectra determined as discussed above. The initialreference spectra may be stored within an array or a matrix comprising aplurality of known signals, including broadband and narrowband signalscorresponding to fluorophores in the image. For example, each column ofa reference matrix may represent a reference spectrum corresponding to aparticular stain or fluorophore. The matrix may be applied to unmix theimage to enable extraction of one or more signals using a linearspectral unmixing process. A spectral signature of a single pixel in themulti-channel image is obtained as a linear combination of the spectralsignatures of all the different fluorophores, each signature beingweighted by the corresponding weight of each fluorophore at that pixel.In a multi-channel image, there may not be any access to the individualweight for each stain's combination; however, the spectral signature ofeach pixel may be retrieved. The set of spectral or target signalsretrieved may be reconstructed to generate an image that is free fromany noisy or unwanted spectra, and consequently fit for analysis.Moreover, a broadband channel unmixed from the image may be processed toestimate optimized reference spectra as described herein.

A structure detection module 115 may be executed to perform operationsincluding detecting and segmenting structures such as nuclei from abroadband unmixed channel, tissue structures, and boundaries thereof.The structure detection operations include nuclear detection, andsegmentation. For nuclear detection, a radial symmetric voting methodmay be applied to determine locations of nuclei within the image orselected region. A gradient magnitude may be computed from the unmixedresult of the spectral unmixing process, and each pixel around aspecified magnitude may be assigned a number of votes that is based on asummation of the magnitude within the region around the pixel. Theformula for computing the gradient magnitude is given as

${\left\lbrack \frac{\partial I}{\partial x} \right\rbrack^{2} + \left\lbrack \frac{\partial I}{\partial y} \right\rbrack^{2}},$where I is the image and

$\frac{\partial I}{\partial x}$is the gradient in the x direction while

$\frac{\partial I}{\partial y}$is the gradient in the y direction. A mean shift clustering operationmay be performed to find the local centers within a voting image, whichrepresents the actual location of the nucleus. A nuclear segmentationoperation uses the now-known centers of the nuclei to performmorphological operations and local thresholding to segment each entirenucleus. Model based segmentation and other operations may also beperformed, depending on a processing power and time requirements. Forexample, the nuclear segmentation operation may also involve learning ashape model of the nuclei from a training data set and using that as theprior knowledge to segment the nuclei in the testing image.

A ranking module 117 is executed to perform operations includingdetermining a confidence level for each detected structure based on aplurality of factors, including shape and intensity information andbiological criteria. These criteria may include a requirement that thestructure or selected region of the image is not co-located with quantumdots or other biomarkers besides the broadband signal. In other words, apure broadband signal is preferred, with the signal being identified asdiscussed above, i.e. based on matching with similar known referencespectra for DAPI, and/or a ubiquitous dispersion throughout the image orregion of the image. Structures such as nuclei that overlap or co-locatewith other stained structures such as tumor or tissue markers may berejected or lowly ranked by examining the pixel intensities. Some nucleimay be rejected. For example, nuclei that are smaller or larger than agiven threshold, or extremely elongated as determined by a ratio betweenlong axes and short axes are rejected. An exemplary threshold for theratio may be 2, i.e. any ellipse having a long axis that is greater than2× larger than a short axis is rejected. An ellipse fitting operationmay be executed to fit an ellipse on top of each segmented nucleus, witha Dice coefficient of the ellipse and the segmented nuclei being used toscore the nuclei with a confidence level, as further described herein.The detected structures may be ranked in order of their scores orconfidence levels as determined by their Dice coefficients and othercriteria. For instance, a subset of the detected nuclei that are scoredwith a confidence level higher than a threshold may be accepted as truepositives.

As described herein, reference spectra estimated from the highest-rankedstructures, by reference spectra estimation module 118, may be used toupdate a reference matrix, with the updated reference matrix being usedin the unmixing operations or subsequent unmixing operations. Referencespectra estimation module 118 observes broadband signatures from, forexample, regions of the image corresponding to the selected or highestquality structures, and estimates reference spectra for these regions.One or more broadband signals observed from these structures and theirsurrounding regions may be used as values in an updated reference matrixthat may be used in a subsequent unmixing operation performed byspectral unmixing module 114. For instance, a value of the referencespectra estimated from a highly-ranked nucleus (i.e., a nucleusidentified as having a high confidence value) may replace the initialDAPI and/or AF reference spectra in a reference matrix. A subsequentunmixing operation using the generated reference matrix therefore yieldsimproved results. The reference spectra estimation may be performed fora plurality of structures in the image or region of the image and medianpixel values of the highly confident structure regions in themulti-spectral image may be used to determine the updated referencespectra. The updated reference spectra may be added as a value to areference matrix stored in database 113 and used in subsequent unmixingoperations.

Moreover, the reference spectra may be generated or updated iterativelyuntil an optimized reference matrix is achieved. Reference matrixoptimization module 119 determines whether or not an updated referencematrix is optimized. For instance, the reference matrix may beiteratively updated via repeated cycles of unmixing 114 and referencespectra estimation 118, with each iteration resulting in a moreoptimized reference matrix for the image or region of the image. Theoptimization may be completed upon determination of a convergence orstabilization of the reference matrix. Alternatively or in addition, anexecution of a threshold number of iterations may be monitored todetermine optimization. The optimized reference matrix yields improvedunmixing results of the hyper-spectral data that are superior whencompared to other methods. For instance, subsequent unmixing operationsusing optimized DAPI reference spectra stored in database 113 maygenerate sharper contours between nuclei and surrounding tissue.Moreover, the optimization may be repeated for different regions of theimage, with each region having a separate optimized reference spectra.

As described above, the modules include logic that is executed byprocessor 105. “Logic”, as used herein and throughout this disclosure,refers to any information having the form of instruction signals and/ordata that may be applied to affect the operation of a processor.Software is one example of such logic. Examples of processors arecomputer processors (processing units), microprocessors, digital signalprocessors, controllers and microcontrollers, etc. Logic may be formedfrom signals stored on a computer-readable medium such as memory 103,which includes including random access memory (RAM), read-only memories(ROM), erasable/electrically erasable programmable read-only memories(EPROMS/EEPROMS), flash memories, etc. Logic may also comprise digitaland/or analog hardware circuits, for example, hardware circuitscomprising logical AND, OR, XOR, NAND, NOR, and other logicaloperations. Logic may be formed from combinations of software andhardware. On a network, logic may be programmed on a server, or acomplex of servers. A particular logic unit is not limited to a singlelogical location on the network.

FIG. 2 shows a method for adaptive spectral unmixing, according to anexemplary embodiment of the present subject disclosure. The method ofFIG. 2 may be performed by a computer executing modules similar to thosedepicted in FIG. 1. The method begins with an image of a specimen orimage data that has been received from a source such as a fluorescencemicroscope associated with or including a scanner or spectral camera(S220), or any source that can capture image content at a range offrequencies, enabling hyperspectral or fluorescence imaging wherein theimage energy is captured at multiple frequencies. The specimen may bestained by means of application of one or more different stainscontaining, for example, quantum dots or fluorophores that areilluminated by a light source. Subsequent to the staining, an image iscaptured by a detection device, for example, a spectral camera, asdescribed above. The image is supplied to a computer that executeslogical instructions stored on a memory for performing the operationsdescribed in the exemplary method.

For instance, the image or regions thereof may be unmixed to retrieve abroadband signal, such as DAPI, from the image (S221). The unmixing mayutilize a non-negative least-squares method as shown herein, forseparating the component fluorescent channels of the image or regionthereof. The image may be delineated into one or more regionsautomatically or manually. This enables subsequent unmixing operationsto be performed on the same or different regions of the image, enablingefficient processing of multiplex images. The initial reference spectramay be retrieved from a control image, or may be estimated from theimage under observation as described herein. The initial referencespectra may be values in a reference matrix containing a plurality ofreference signals that are applied to the observed mixture of signals,resulting in a plurality of unmixed component signals of the image. Forinstance, a matrix populated with reference spectra and a plurality ofnarrowband signals corresponding to the target signals may be applied tothe mixture of signals, resulting in a set of component signals.

To highlight an exemplary non-negative least squares unmixing operation,let A_(m×n) be the endmember reference spectra matrix, with each columnof A corresponding to an endmember spectrum that is a m-dimensionalvector. A=[a_(d), a_(a), a_(1 . . . , k)] are the spectra for DAPI, AFand quantum dots respectively. n is the number of biomarkers/endmembers.Y_(m×N) is the multi-spectral image containing N pixels, with each pixelbeing a m-dimensional vector imaged from m spectral narrow-band filters.The unmixing equation shown below is formulated to minimize the overallleast square errors:

$\min\limits_{x}{\sum\limits_{i = 1}^{N}{{{{Ax}_{i} - y_{i}}}_{2}^{2}\mspace{14mu}\left( {{{subject}\mspace{14mu}{to}\mspace{14mu} x_{i}} \geq 0} \right)}}$with x_(i) being a n-dimensional vector of abundances corresponding tothe i^(th) pixel of the unmixed images and y_(i) being the i^(th) pixelof Y. Solving the equation yields the unmixed images for the nendmembers X_(n×N)=[x₁, x₂, . . . , x_(N)]. More details about thenon-negative least squares solution process may be found in Lawson, C.L. and R. J. Hanson, Solving Least Squares Problems, Prentice-Hall,1974, Chapter 23, p. 161.

Among the resultant set of component signals is at least one broadbandsignal that is analyzed for a presence of structures (S223). This stepincludes detecting and segmenting structures such as nuclei from the atleast one broadband channel data of the image, as well as identifyingtissue structures, and boundaries thereof. The structure detectionoperations include nuclear detection, and segmentation. For nucleardetection, a radial symmetric voting method may be applied to determinelocations of nuclei within the image or selected region. An imagemagnitude may be computed from the unmixed result of the spectralunmixing process, and each pixel around a specified magnitude may beassigned a number of votes that is based on a summation of the magnitudewithin a region around the pixel. A mean shift clustering operation maybe performed to find the local centers within a voting image, whichrepresents the actual location of the nucleus. Nuclear segmentation usesthe now-known centers of the nuclei to perform morphological operationsand local thresholding to segment the entire nucleus. Model basedsegmentation and other operations may also be performed, depending on aprocessing power and time available.

Noise in the image and a potentially inaccurate reference spectrum maylead to false positive results among detected structures. A rankingoperation is therefore executed to determine a confidence level for eachdetected structure based on a plurality of factors, including shape andintensity information and biological criteria. These criteria include arequirement that the structure or selected region of the image is notco-localized with quantum dots by checking the signal existence at thesame location for both DAPI and quantum dot images, or with otherbiomarkers besides the broadband signal. Structures that are smaller orlarger than a given threshold, or extremely elongated as determined bythe ratio between long axes and short axes are rejected. An ellipsefitting operation may be executed to fit an ellipse on top of eachsegmented nucleus, with a Dice coefficient of the ellipse being used toscore the nuclei with a confidence level. The detected nuclei may beranked in order of their scores or confidence levels as determined bytheir Dice coefficients and other criteria. For instance, a subset ofthe detected nuclei that are scored with a confidence level higher thana threshold may be accepted as true positives.

Reference spectra may be estimated from the highest-ranked structures(S225). This step includes observing broadband signatures such as DAPIfrom the highest-ranked structures within pre-defined regions of theimage, and estimating updated reference spectra for these structures andtheir surrounding regions. The retrieved signals may be used to updateor generate a reference matrix (S227) that may be used in a subsequentunmixing operation. For instance, a value of the updated referencespectrum estimated from a highly-ranked nucleus may replace the initialDAPI reference spectrum in an updated reference matrix. An unmixingoperation using the generated reference spectra matrix/data or asubsequent unmixing operation using the updated reference matrixtherefore yields improved results. The reference spectra estimation(S225) may be performed for a plurality of structures in the image orregion of the image and a median broadband signal may be used todetermine the updated reference spectra (S227). The updated referencespectra may be added as a value to a reference matrix stored in areference database and used in subsequent unmixing operations.

Moreover, the reference spectra may be updated iteratively until anoptimized reference matrix is achieved (S229). A determination thatchecks if a maximum number of iterations is exceeded or whether anupdated reference matrix is optimized may trigger an additional unmixing(S221) process, followed by structure detection (S223), referencespectra estimation (S225), and updating (S227), with each iterationresulting in an improved reference matrix for the image or region of theimage. The optimization (S229) may be completed upon determination of aconvergence or stabilization of the reference matrix. A convergence maybe determined by monitoring a relative difference between each updatedreference spectra between two consecutive iterations. Alternatively orin addition, an execution of a threshold number of iterations may bemonitored to determine optimization, with an exemplary threshold numbert=100. An example loop for optimal reference spectra optimization (S229)may include the following operations:

1: Input initial reference spectra matrix A = [a_(d), a_(a),a_(1 . . . , k)] and multi- spectral image Y 2: for t ≤ Max IterationNumber or ∥A_(t+1) − A_(t)∥² ≤ ϵ do 3:$\left. {{Solve}\mspace{14mu}{for}\mspace{14mu} x\text{:}\mspace{14mu}{\min\limits_{x}\sum\limits_{i = 1}^{N}}} \middle| {{Ax}_{i} - y_{i}}|_{2}^{2}\mspace{14mu}\left( {{{subject}\mspace{14mu}{to}\mspace{14mu} x_{i}} \geq 0} \right) \right.\;$4: Estimate the broad-band reference spectra a_(d) and a_(a) for DAPIand AF (Step S225 in FIG. 2) and update A_(t) using A_(t+1) = [a_(d)^(new), a_(a) ^(new), a_(1 . . . , k)] 5: end for

The optimized reference matrix yields reliable unmixing of thehyper-spectral data that are improved when compared to other methods.Moreover, the optimization may be repeated for different regions of theimage, with each region having a separate optimized reference spectra.The method ends when all required reference spectra in a referencespectra matrix are optimized (S229) and the image can undergo furtherprocessing.

FIGS. 3A and 3B show an image 330 divided into regions for adaptivespectral unmixing, according to an exemplary embodiment of the presentsubject disclosure. Image 330 may be depicted on a display of acomputer, and may be generated by an application such as one of themodules described in FIG. 1. Image 330 shows a specimen scanned from aslide, and depicts a plurality of structures 331. The specimen may, forexample, take the form of a tissue section obtained from a human oranimal subject, such as a formalin-fixed, paraffin-embedded tissuesample. The specimen may be living cellular tissue, frozen cells, tumorcells, blood, throat culture, or other; the type or nature of specimenis not particularly important. The structures may be tissue cells,nuclei, cancer cells, or any other structure.

As described herein, the image may be delineated into one or moreregions, enabling parallel unmixing and optimizing operations to beperformed on different regions. This may be achieved by overlaying asparse grid over image 330, as represented by dashed lines 333 depictedin FIG. 3A. Structures within a distance of one or more grid vertices,such as nucleus 331 in region 335 may be sampled for estimated referencespectra. This enables independent processing of separate portions of theimage to estimate median reference spectra for the entire image.Alternatively or in addition, the regions delineated by lines 333 may beprocessed independently to determine separate optimized referencespectra for each region.

Moreover, as shown in FIG. 3B, custom regions may be defined based onstructures or features observed in the image. This custom segmentationenables generation of separate optimized reference spectra per region,particularly in the case of images having fluctuating DAPI channels. Thecustom segmentation may be determined automatically based onidentification of structures in the image, i.e. using variation inintensities of pixels, tissue heterogeneity, etc. For instance, nucleiwithin a tumor region may be larger, triggering a separation of a tumorregion from other regions of the image. A local histogram may be used togroup and delineate similar regions, with subsequent detection andoptimization processes being performed independently per region. Customregions may be user-selectable via a user interface. For instance, aDAPI or broadband region may be identified and selected by a user.Moreover, the region selection may occur anytime during the process,including after an identification of high-confidence structures.

FIG. 4 shows a method for detecting structures in a broadband signal ofan image, according to an exemplary embodiment of the present subjectdisclosure. The method of FIG. 4 may be performed by a computerexecuting modules similar to those depicted in FIG. 1. The method beginswith an initial unmixing operation (S440) using an initial referencespectra, resulting in at least one unmixed broadband signal. A region ofthe image may be selected (S441) for nuclei detection (S445) andsegmentation (S447). For instance, the image may be delineated into oneor more regions automatically or manually, as described herein.Structures within a distance of one or more grid vertices may be sampledfor estimated reference spectra, as shown in FIG. 3A. Upon selection ofthe image region, morphological operations may be performed (S443) toremove background noise, and to determine foreground of nuclei. Thecenter of each nucleus may be determined using methods such as radialsymmetry voting (S445). For instance, given the unmixed broadbandchannel, an image magnitude is computed from the channel, and one ormore votes at each pixel are accumulated by adding the summation of themagnitude within a selected region. Mean shift clustering may be used tofind the local centers in the region, with the local centersrepresenting actual nuclear locations.

The detected nuclei may be further segmented from surrounding tissue(S447). Segmentation uses the centers of the nuclei determined in S445to perform morphological operations (such as dilation to expand theregion and erosion to remove the isolated noise and shrink the region)on a region adjacent the nuclei to obtain the surrounding tissue. Localthresholding may be performed to segment the entire nucleus. As theintensities of the nuclear pixels are higher than those in itsneighborhood, an intensity cut-off/threshold can be used to separate thenuclei and neighborhood background regions. Model based segmentation andother operations such as pixel-wise classification using K-mean methodmay also be performed. With the nuclei and surrounding tissues beingdetected and segmented, an ellipse fitting operation (S449) may beperformed on each nucleus to enable subsequent confidence determinationand ranking operations. Since most nuclei are round and/or elliptical inshape, a robust ellipse fitting algorithm based on random sampleconsensus (RANSAC) may be executed to fit an ellipse on top of eachsegmented nucleus. Upon fitting an elliptical mask on each nucleus,subsequent confidence level determination operations may be executed. Anexample ellipse fitting method is as follows:

Let {right arrow over (x_(i))}=(x_(i),y_(i)) be the landmarks along theboundary of the nucleus. Minimize the sum of squared algebraic distancesd({right arrow over (a)})=Σ_(i=1) ^(N){right arrow over (a)}·{rightarrow over (x_(i))} and solve for the parameters of the ellipse model{right arrow over (a)}, where {right arrow over (a)}·{right arrow over(x_(i))}=ax²+b×y+cy²+ey+f, and {right arrow over (a)}=(a, b, c, d, e, f)representing parameters of the ellipse.

FIG. 5 shows a method for determining a confidence level of detectedstructures, according to an exemplary embodiment of the present subjectdisclosure. The confidence level may be based on a plurality of factors,including shape and intensity information and biological criteria. Themethod begins with a result of a nuclear component detection processS550 as described above, with detected nuclei being segmented and fittedwith elliptical masks. Each nucleus is observed independently, with anellipse being selected (S551) prior to performing confidence leveldeterminations. For instance, an area of the ellipse is computed (S553),and if the area is larger or smaller than a threshold, the ellipse maybe rejected (S559). This operation also includes determining anelongation of the ellipse, i.e. a ratio of a long axis to a short axisof the ellipse, with any ratio being above or below a thresholdresulting in the ellipse being rejected (S559). For example, a maximumthreshold value may be 2. If the area and elongation of the ellipse meetthe threshold standards, then the ellipse fit is scored (S555) using aDICE coefficient as shown below:

${DICE} = {\frac{2{{{V(p)} - {V(q)}}}}{{V(p)} + {V(q)}}\mspace{14mu}\frac{2{{{V(p)}\bigcap{V(q)}}}}{{V(p)} + {V(q)}}}$where V is an area operator, p is a binary image of the segmentednucleus, and q is the elliptical mask.

The nucleus may be scored and ranked (S555) based on the DICEcoefficient, and scores lower than a threshold may be rejected (S559).In one exemplary embodiment, a threshold DICE ratio is 0.9. Ellipsesmeeting this threshold (i.e. scoring 0.9 or higher) may be subject toadditional criteria (S557) before being approved, such as meetingbiological constraints. These may include requiring that the detectedstructure is not co-located with quantum dots or other biomarkersbesides the broadband signal, or does not overlap with other stains. Ifthese conditions are not met, the ellipse may still be rejected (S559).If these constraints are met, then the method determines if there areadditional ellipses within the region to be scored (S558). If there areadditional ellipses, the method starts again with the selection of thenext ellipse (S551). If there are no more ellipses, the method cancontinue to perform additional operations such as ranking and estimatingthe reference spectra from the accepted structures.

FIG. 6 shows a nuclear ranking using ellipse fitting, according to anexemplary embodiment of the present subject disclosure. Image 630depicts a specimen scanned from a slide comprising a plurality ofstained structures 631. For example, the specimen may include a tissuesection, such as a formalin-fixed, paraffin-embedded tissue specimen,and may have been obtained from a human or animal subject. The specimenmay be living cellular tissue, frozen cells, tumor cells, blood, throatculture, or other; the type or nature of specimen is not particularlyimportant. The structures may be tissue cells, nuclei, cancer cells, orany other structure. An ellipse 637 is fit over a structure 631, andeach ellipse may be scored and ranked based on a plurality of thresholdsthat are applied to the ellipse, as described herein. For instance,ellipses that have a DICE coefficient of less than 0.9 may be rejected,as marked with an “R” in FIG. 6. Ellipses that score higher than 0.9 andthat meet additional biological criteria may be accepted and marked with“A”. Further, an elongation of an ellipse may be subject to thresholds,with extremely elongated ellipses being rejected, based on an assumptionthat most nuclei are close to spherical in shape. Intensity constraintsmay also be applied to the detected nuclei prior to acceptance orrejection. For example, if the intensity of the nuclei pixels is lessthan 0.1 determined by mean intensity−2*standard deviation of intensity,then the nucleus may not be accepted.

Moreover, tissue auto-fluorescence regions are generally found in theneighborhood of the nuclei. With the high confidence nuclei being markedas accepted, a small neighborhood within a few pixels around a nucleusis designated as the tissue region, as shown in by ring 638 around thefitted ellipse 637. As the nuclear ranking procedure ensures thesegmentation of nuclei and tissue regions, broadband spectra includingDAPI and AF may be estimated from these regions, with reference spectramatrix A being generated or updated accordingly. Subsequent unmixing andestimation steps described above may be recursively applied until thebroad-band reference spectra stabilize or the maximum number ofiterations is exceeded. Median reference spectra for the image or forselected regions of the image may be stored in an optimized referencespectra matrix.

FIG. 7 shows a graph identifying a median reference spectrum estimatedfrom a plurality of highly-ranked structures, according to an exemplaryembodiment of the subject disclosure. Each single nucleus is estimatedto have a reference spectra identified by dashed lines 760. For aspecific region of the image, a median of the reference spectra for eachnucleus within that region is computed and depicted as solid line 761which is the median of all the dashed lines 760. This is used as thegenerated reference spectra for that region in an updated referencematrix A. Alternatively or in addition, FIG. 7 may represent a pluralityof reference spectra 760 determined for a corresponding plurality ofiterations of the optimization process, with the resulting medianreference spectrum 761 representing the optimized reference spectrum.The optimized reference spectra are used to properly unmix the image,resulting in improved results.

FIG. 8 shows a comparison of nuclear detection results using initialreference spectra versus updated reference spectra, according toexemplary embodiments of the subject disclosure. Resulting images 871shown in the left column depict nuclei detected in a broadband channelthat is unmixed using a standard reference spectra derived from acontrol image. Results 872 in the right column show nuclei detectedafter an adaptive unmixing process as described in the disclosedembodiments. It is evident that in each region, there are improvednuclear detections using optimized reference spectra versus usingstandard reference spectra.

The disclosed image analysis operations such as improvements inestimation of the broadband reference spectra obtained using improvedidentification of cells minimize differences between the observedspectral image or hyperspectral image datasets and the estimated matrixperformed by existing methods, while obtaining estimates of thebroadband signatures such as AF and DAPI particular to the image. Otherbroadband signatures, for example, those of red blood cells may also beoptimized. The disclosed systems and methods are highly suitable forvessel detection and macrophage segmentation, as the methods willprovide more accurate spectral unmixing results and sharper and cleanerDAPI channels. Both vessel detection and macrophage segmentation requirean accurate estimation of the nuclei (stained by DAPI) as the firststep. The method of the present invention is also applicable to in situhybridization (ISH) images. ISH is a useful technique for spatiallylocalizing certain probes/targets within tissues and cells, whichprovide information about gene expression and genetic loci. The probescan be in the form of fluorescent markers (as in FISH) or chromogenicmarkers (as in CISH). For both cases, the adaptive unmixing method ofthe present invention may be applied to FISH images or CISH images toobtain, for example, improved estimates of a nuclear marker's signature.Due to staining variations, the signature for a nucleus can vary and abest estimate of this nucleus channel can differ significantly perimage. However, the adaptive unmixing method of the present invention,which may be for DAPI estimates in fluorescent images, is also used toidentify a group of top ranked shaped cells in FISH and CISH images.Further, a statistical averaging over these shaped cells (for examplecomputing a median of the reference spectrum of the top ranked shapedcells, in accordance with the present invention) gives an improvedestimate of the nucleus reference spectra or reference colors. As such,the adaptive umixing method of the present invention, which usesphysically meaningful structures to obtain more reliable estimates ofreference vectors, is applicable for both brightfield and fluorescentimages, for example chromogenic brightfield or darkfield IHC images,chromagenic ISH brightfield or darkfield images, fluorescent ISH images,and/or quantum dot images. Moreover, besides medical applications suchas anatomical or clinical pathology, prostate/lung cancer diagnosis,etc., the same methods may be performed to analyze other types ofsamples such as remote sensing of geologic or astronomical data, etc.Images may be further refined by eliminating known or obvious sources ofnoise by, for instance, being compared to known or ideal sets of signalsfrom similar materials. Other refinement processes include adjusting aminimum or a maximum of intensities to highlight a specific range andeliminating signals outside the range, adjusting a contrast to see amore dynamic range, and other imaging operations. For large or multipleslide/image analysis, or for analyzing one or more image cubes, theoperations described herein may be ported into a hardware graphicsprocessing unit (GPU), enabling a multi-threaded parallelimplementation.

The foregoing disclosure of the exemplary embodiments of the presentsubject disclosure has been presented for purposes of illustration anddescription. It is not intended to be exhaustive or to limit the subjectdisclosure to the precise forms disclosed. Many variations andmodifications of the embodiments described herein will be apparent toone of ordinary skill in the art in light of the above disclosure. Thescope of the subject disclosure is to be defined only by the claimsappended hereto, and by their equivalents. Further, in describingrepresentative embodiments of the present subject disclosure, thespecification may have presented the method and/or process of thepresent subject disclosure as a particular sequence of steps. However,to the extent that the method or process does not rely on the particularorder of steps set forth herein, the method or process should not belimited to the particular sequence of steps described. As one ofordinary skill in the art would appreciate, other sequences of steps maybe possible. Therefore, the particular order of the steps set forth inthe specification should not be construed as limitations on the claims.In addition, the claims directed to the method and/or process of thepresent subject disclosure should not be limited to the performance oftheir steps in the order written, and one skilled in the art can readilyappreciate that the sequences may be varied and still remain within thespirit and scope of the present subject disclosure.

The invention claimed is:
 1. A non-transitory computer-readable mediumstoring computer-executable instructions which, when executed by one ormore processors, cause the one or more processors to perform operationscomprising: accessing an image of a biological sample, wherein the imagecomprises a mixture of signals; utilizing an initial reference vector tounmix a first region of the image to generate an unmixed first region ofthe image; processing the unmixed first region of the image using asegmentation operation to detect a plurality of structures within theunmixed first region, wherein each structure of the plurality ofstructures corresponds to a tissue type; generating an updated referencevector for the unmixed first region based on the plurality ofstructures; and utilizing the updated reference vector to unmix theimage.
 2. The non-transitory computer-readable medium of claim 1,wherein the initial reference vector is retrieved from a control image.3. The non-transitory computer-readable medium of claim 2, wherein theplurality of structures are detected via radial symmetric voting and/orvia segmentation between a plurality of nuclei and a plurality of tissuecells.
 4. The non-transitory computer-readable medium of claim 2,wherein the operations further comprise ranking the plurality ofstructures based on biological criteria comprising at least on onelocation in a pure 4′,6-diamidino-2-phenylindole (DAPI) region, ahematoxylin region, a non-overlapping nucleus, a size threshold and anellipse fit.
 5. The non-transitory computer-readable medium of any ofclaim 4, wherein the operations further comprise determining aconfidence level for each of the plurality of structures.
 6. Thenon-transitory computer-readable medium of claim 5, wherein theoperations further comprise estimating the updated reference vector froma subset of the plurality of structures that meets a confidence levelthreshold.
 7. The non-transitory computer-readable medium of claim 6,wherein the confidence level threshold is based in part on a DICE ratioassociated with the ellipse fit.
 8. The non-transitory computer-readablemedium of claim 6, wherein the operations further comprise estimatingthe updated reference vector from a region adjacent to one or morestructures of the subset of the plurality of structures.
 9. Thenon-transitory computer-readable medium of claim 1, wherein theoperations further comprise repeatedly unmixing the image and updatingthe initial reference vector, until an optimal reference spectrum isachieved.
 10. The non-transitory computer-readable medium of claim 1,wherein the mixture of signals comprises at least one of a4′,6-diamidino-2-phenylindole (DAPI) signal, an autofluorescence signal,and a hematoxylin (HTX) signal.
 11. A system for unmixing an image,comprising: one or more processors; and at least one memorycommunicatively coupled to the one or more processors, the at least onememory storing computer-executable instructions that, when executed bythe one or more processors, cause the one or more processors to performoperations comprising: accessing an image of a specimen; identifying,from the image, an unmixed signal corresponding to a stain; detecting aplurality of structures by analyzing the identified unmixed signal;estimating a reference vector for the stain based on a subset of theplurality of structures; unmixing the image using the estimatedreference vector; and repeating the identifying, detecting, estimating,and unmixing steps for a predetermined number of iterations or until anoptimal reference vector is obtained.
 12. The system of claim 11,wherein said plurality of structures comprise a plurality of nuclei. 13.The system of claim 12, wherein said the subset of the plurality ofstructures comprise a subset of the plurality of nuclei that meets aconfidence level threshold.
 14. The system of claim 13, wherein theoperations further comprise delineating the image into a plurality ofregions, and wherein the subset of the plurality of nuclei is locatedwithin a first region of the plurality of regions.
 15. The system ofclaim 14, wherein the operations further comprise unmixing a secondregion of the plurality of regions using the reference vector.
 16. Amethod comprising: unmixing an image comprising a mixture of signals togenerate one or more unmixed signals; detecting a plurality ofstructures in the unmixed signals; ranking the plurality of structuresin order of a confidence level; estimating a reference vector for atleast one signal in the mixture of signals based at least on a subset ofstructures among the plurality of structures, wherein the subset ofstructures have a confidence level higher than a threshold confidencelevel; storing the reference vector in a reference matrix associatedwith the image; and subsequently unmixing the image using the referencematrix.
 17. The method of claim 16, further comprising repeating thedetecting, ranking, estimating, and subsequent unmixing until an optimalreference vector is obtained.
 18. The method of claim 16, whereinsubsequently unmixing the image using the reference matrix includesunmixing a region of the image using the reference vector.
 19. Themethod of claim 16, wherein the reference vector is estimated by using amedian signal of the mixture of signals.
 20. The method of claim 16,wherein the confidence level is determined based on shape and intensitydata corresponding to a structure of the plurality of structures.